Synthetic Gradient Methods with Virtual Forward-backward Networks
نویسندگان
چکیده
The concept of synthetic gradient introduced by Jaderberg et al. (2016) provides an avant-garde framework for asynchronous learning of neural network. Their model, however, has a weakness in its construction, because the structure of their synthetic gradient has little relation to the objective function of the target task. In this paper we introduce virtual forward-backward networks (VFBN). VFBN is a model that produces synthetic gradient whose structure is analogous to the actual gradient of the objective function. VFBN is the first of its kind that succeeds in decoupling deep networks like ResNet-110 (He et al., 2016) without compromising its performance.
منابع مشابه
Using Feedforward Neural Networks and Forward Selection of Input Variables for an Ergonomics Data Classification Problem
A method was developed to accurately predict the risk of injuries in industrial jobs based on datasets not meeting the assumptions of parametric statistical tools, or being incomplete. Previous research used a backward-elimination process for feedforward neural network (FNN) input variable selection. Simulated annealing (SA) was used as a local search method in conjunction with a conjugate-grad...
متن کاملMean Field Residual Networks: On the Edge of Chaos
We study randomly initialized residual networks using mean field theory and the theory of difference equations. Classical feedforward neural networks, such as those with tanh activations, exhibit exponential behavior on the average when propagating inputs forward or gradients backward. The exponential forward dynamics causes rapid collapsing of the input space geometry, while the exponential ba...
متن کاملProjected-gradient algorithms for generalized equilibrium seeking in Aggregative Games are preconditioned Forward-Backward methods
We show that projected-gradient methods for the distributed computation of generalized Nash equilibria in aggregative games are preconditioned forward-backward splitting methods applied to the KKT operator of the game. Specifically, we adopt the preconditioned forward-backward design, recently conceived by Yi and Pavel in the manuscript “A distributed primal-dual algorithm for computation of ge...
متن کاملForward-backward Splitting in Deformable Image Registration: a Demons Approach
Efficient non-linear image registration implementations are key for many biomedical imaging applications. By using the classical demons approach, the associated optimization problem is solved by an alternate optimization scheme consisting of a gradient descent step followed by Gaussian smoothing. Despite being simple and powerful, the solution of the underlying relaxed formulation is not guaran...
متن کاملBackward-forward Algorithms for Structured Monotone Inclusions in Hilbert Spaces
In this paper, we study the backward-forward algorithm as a splitting method to solve structured monotone inclusions, and convex minimization problems in Hilbert spaces. It has a natural link with the forward-backward algorithm and has the same computational complexity, since it involves the same basic blocks, but organized differently. Surprisingly enough, this kind of iteration arises when st...
متن کامل